On the PAC-Bayes Bound Calculation based on Reproducing Kernel Hilbert Space

PAC-Bayes risk bound combining Bayesian theory and structure risk minimization for stochastic classifiers has been considered as a framework for deriving some of the tightest generalization bounds. A major issue for calculating the bound is the unknown prior and posterior distributions of the concept space. In this paper, we formulated the concept space as Reproducing Kernel Hilbert Space (RKHS) using the kernel method. We further demonstrated that the RKHS can be constructed using the linear combination of kernels, and the support vectors and their corresponding weights of SVM outputs describe the complexity of concept space. Therefore the calculation of PAC-Bayes bound can be simulated by sampling weights of support vectors in RKHS. The experimental results using random and Markov Chain Monte Carlo (MCMC) samplings showed that the simulation is reasonable and effective in practice.